Development of a heart failure filter for Medline: an objective approach using evidence-based clinical practice guidelines as an alternative to hand searching

BACKGROUND Heart failure is a highly debilitating syndrome with a poor prognosis primarily affecting the elderly. Clinicians wanting timely access to heart failure evidence to provide optimal patient care can face many challenges in locating this evidence. This study developed and validated a search filter of high clinical utility for the retrieval of heart failure articles in OvidSP Medline. METHODS A Clinical Advisory Group was established to advise study investigators. The study set of 876 relevant articles from four heart failure clinical practice guidelines was divided into three datasets: a Term Identification Set, a Filter Development Set, and a Filter Validation Set. A further validation set (the Cochrane Validation Set) was formed using studies included in Cochrane heart failure systematic reviews. Candidate search terms were identified via word frequency analysis. The filter was developed by creating combinations of terms and recording their performance in retrieving items from the Filter Development Set. The filter's recall was then validated in both the Filter Validation Set and the Cochrane Validation Set. A precision estimate was obtained post-hoc by running the filter in Medline and screening the first 200 retrievals for relevance to heart failure. RESULTS The four-term filter achieved a recall of 96.9% in the Filter Development Set; 98.2% in the Filter Validation Set; and 97.8% in the Cochrane Validation Set. Of the first 200 references retrieved by the filter when run in Medline, 150 were deemed relevant and 50 irrelevant. The post-hoc precision estimate was therefore 75%. CONCLUSIONS This study describes an objective method for developing a validated heart failure filter of high recall performance and then testing its precision post-hoc. Clinical practice guidelines were found to be a feasible alternative to hand searching in creating a gold standard for filter development. Guidelines may be especially appropriate given their clinical utility. A validated heart failure filter is now available to support health professionals seeking reliable and efficient access to the heart failure literature.